TextCL: A Python package for NLP preprocessing tasks

Preprocessing text data sets for use in Natural Language Processing tasks is usually a time-consuming and expensive effort. Text data, normally obtained from sources such as, but not limited to, web scraping, scanned documents or PDF files, typically unstructured prone to artifacts other types of noise. The goal the TextCL package simplify this process by providing multiple methods suited preprocessing. It includes functionality splitting texts into sentences, filtering sentences language, perplexity filtering, removing duplicate sentences. Another offered outlier detection module, which allows identify filter out that are different main topic distribution set. This method selecting one several unsupervised algorithms, as TONMF (block coordinate descent framework), RPCA (robust principal component analysis), SVD (singular value decomposition) apply it data.